Keeping track of all ongoing colorectal cancer trials using a mobile application: Usability and satisfaction results of the Dutch Colorectal Cancer Group Trials application

Background and Aim: Both the number and complexity of medical trials are increasing vastly. To facilitate easy access to concise trial information, a freely available mobile application including all ongoing clinical trials of the Dutch Colorectal Cancer Group (DCCG) was developed. The aim of this study was to investigate the use and user satisfaction over the first 2 years. Methods: The application was launched in January 2015 on iOS and Android platforms. Google Analytics was used to monitor anonymous user data up to February 2017. In addition, an online survey regarding the use and satisfaction among health-care professionals and research affiliates active in the field of colorectal cancer in the Netherlands was conducted. Results: A total of 6173 unique users were identified, of which 1822 (30%) were from the Netherlands, representing a total of 16,065 and 10,987 (68%) sessions, respectively. The median session duration per day was 01:47 min (IQR 0:51–03:03). The mobile application was mostly used on Monday, Tuesday, and Thursday, and the number of sessions was highest during the following time frames: 12–13 pm (9%), 17–18 pm (9%), and 13–14 pm (8%). Of 121 survey responses, most were medical doctors (47%), nurses (25%), or researchers (9%), working either in a teaching (40%), academic hospital (32%), or general hospital (19%). 83% of all respondents rated the application 4 or higher for satisfaction on a 5-point scale. Highest reported reasons of the use were urgent trial inquiry (57%) and usage during multi-disciplinary meetings (49%). Conclusion: The DCCG Trials application is frequently used, and the majority of users is highly satisfied. Relevance for Patients: Clustering trial information into one platform, such as DCCG trials app, has shown to be useful for medical professionals treating patients with colorectal carcinoma in the Netherlands.


Introduction
The number of registered clinical trial protocols on clinicaltrials. gov has increased from 12,020 in 2005 to over 230,000 in 2017, and the yearly number of newly registered studies is approaching *Corresponding author: Joost Huiskens Department of Medical Oncology, Amsterdam University Medical Center, Meibergdreef 9 1105 AZ, Amsterdam, Netherlands Tel: +31 (20) 56 63 995 Email: j.huiskens@amc.uva.nl 30,000 [1]. In the field of colorectal cancer alone, the third most common cancer worldwide, 4482 trials, was registered by the end of 2017 (Figure 1).
Besides the increasing number of clinical trials, there has been an increase in protocol design complexity during the past decade. The incidence of added trial protocol amendments is growing and trial criteria are increasingly specified and complex [2]. Among many others, these factors lead to the shortage of both knowledge and time for health-care professionals to participate in clinical trials [3,4]. The complexity of clinical trials potentially hampers the inclusion of patients, which is the leading cause of problems in the conduction of clinical trials [5][6][7][8]. Lack of accrual can lead to trial discontinuation or results in an insufficient sample size, both of which challenge the ethics of the exposure of patients to trial treatment which is often outside of routine clinical practice. Evidence on effective strategies to improve the conduction of clinical trials is scarce and mostly limited to the recruitment for randomized trials [9]. The Cochrane systematic review of interventions to improve the trial recruitment included 45 studies, of which only 12 were considered to be of low risk of bias [10]. Opt-out procedures, telephone reminders, and open trial designs might be effective strategies to improve patient recruitment. However, these strategies do not comply with the guidelines of good clinical practice. Interestingly, none of these reported interventions targeted medical doctors, who usually assess patient eligibility for trials and ask patients to participate in clinical trials.
This manuscript illustrates the design of a smartphone application that provides easy to access and up-to-date information on ongoing Dutch clinical trials for patients with colorectal cancer. The aim of this study is to investigate the usability and satisfaction of the application 2 years after its introduction.

Materials and Methods
Anonymous user data of the Dutch Colorectal Cancer Group (DCCG) Trials application were collected between the February 1 2015 and February 1 2017 and analyzed using Google Analytics [11]. In addition, an online survey among health-care professionals and research affiliates active in the field of colorectal cancer was conducted.
The DCCG is a research collaboration in the Netherlands between all medical disciplines involved in the diagnosis and management of colorectal cancer [12]. The DCCG Trials application is a mobile application containing concise information on all the DCCG multicenter trials open for inclusion. The application is freely downloadable for iOS and Android [13]. It can also be accessed through the website: www.trialapp.nl/dccg/ dccg-app/without using the mobile application. Users can find relevant information for the registration of a patient for a trial. Trial coordinators can be e-mailed or called directly through the application for questions. The information provided in the application per trial can be directly updated by the responsible investigators. Trials can be found in two different ways, either by following a decision-tree based on clinical features or by utilizing the search function. Using the decision-tree users can search for trials by answering questions such as; is the tumor located in the colon, rectum, or does the patient have metastatic disease? Every trial page has six buttons: Design, criteria, requirements, latest news, the trial website, and contact information ( Figure 2). Anonymously collected user data by Google Analytics [11] included the number of (new) users, the number of sessions, the bounce rate (users that opened and closed the app immediately), average session duration per user per day, geographical location based on IP address, the number and duration of trial information page visits, the number of times the decision-tree option which was used, and the amount the news option which was used.
To measure usability and satisfaction, an online survey was sent either through the DCCG Trials application, in the DCCG newsletter, or by email using the DCCG mailing list, consisting of email addresses of health-care professionals and research

Results
In its first 2 years, the DCCG Trials application amassed a total of 16,065 sessions and 89,711 page views by 6,173 unique users worldwide. The median session duration per day was 01:47 min (IQR 0:51-03:03). The median number of pages visited per session was 6 (IQR 4-7). In total, the application was used in 102 countries (Supplementary Figure 1).
In the Netherlands, 1,822 (30%) unique users undertook a total of 10,987 sessions that amounted to 68% of all sessions of the application. The median average session duration per day in the Netherlands was 02:00 min (IQR 01:03-03:30) and the bounce rate was 3% which was the lowest of all countries ( Table 1). The mobile application was most used on Monday, Tuesday, and Thursday, and the number of sessions was highest during the following time frames: 12-13 pm (9%), 17-18 pm (9%), and 13-14 pm (8%).
Trial information pages were visited 15,896 times and the median time on a trial page ranged from 9 s to 47 s. The most frequently visited study pages are depicted in Table 2.

Online survey
Of a total of 121 respondents, 76 (63%) answered that they have used the application, whereas 45 (37%) did not. Reasons for not using the application were unawareness of its existence (n = 26, 59%), preference for other resources (n = 8, 18%), or that the responder is not provided a smartphone at work (n = 4, 9%) ( Figure 3).

Discussion
The present study illustrates that a smartphone application with concise information on multicenter trials on colorectal carcinoma is often used and that reported satisfaction with the application is high. More than 70% of the users are medical doctors or nurses, and the application is mostly used during multidisciplinary meetings and during work in the outpatient clinic. The DCCG Trials application was launched as an experimental tool that provides concise information on clinical trials which might benefit patient inclusion. This application also offers insights for implementation in other clinical fields.
Since the DCCG is based in the Netherlands, it was to be expected that most users derive from the Netherlands. The increased Dutch user duration rate supports the suggestion that the  application is a useful tool when professionals encounter potential inclusions. The survey illustrated that the majority of the users were physicians in teaching hospitals [14]. Compared to academic centers, these hospitals often have a higher patient turnover but lower awareness of ongoing trials, which could be a potential cause of slow patient enrollment. The application directly offers information on inclusion and exclusion criteria, increasing the chances of patient inclusion. In addition, patient inclusion often has a certain momentum in clinical practice. If eligibility can directly be assessed by use of the application or when a potential trial candidate can directly be discussed with the trial coordinator, likelihood of patient enrollment increases. Finally, the option to directly update users with new trial information reminds users of the existence of a trial, indirectly increasing the likelihood of including patients in trials. This is the first study presenting data of an application with the goal to improve clinical trial conduction and patient recruitment. Many applications have been developed in which trial information can be found [15][16][17][18]. However, no data are available about the effect of these applications on the conduction and recruitment of clinical trials. Little evidence is available for any intervention on the effect of patient recruitment in clinical trials [10]. It is difficult to prove the effect of the DCCG Trials application on trial enrollment. However, the fact that the application is frequently used by relevant users suggests that an application with relevant information on ongoing trials could have a positive impact on trial enrollment.
The current analyses have several limitations. First, presenting data on the duration of sessions and trial page visits does not necessarily reflect a positive user experience or positive effect on the conduction of a particular trial. Preferably, the effect of the application on inclusion rates is investigated, but the available data and trials were limited and a direct causal effect of the application on inclusion rates would be difficult to assess. Second, the survey was made available online, and respondents were recruited through the app, DCCG newsletter, and DCCG mailing list, which could have introduced a selection bias. It can be assumed that users that are very positive about the application are more willing to submit a survey, yet it could also be assumed that users with a very negative user experience are motivated to express their opinion.
Currently, the application is not available on iOS devices due to new technical requirements which are not met by the current version. The application is still available on Android devices. All DCGG studies from the application can be found in the trial app application which is available on all devices.

Conclusion
We report that our smartphone application with study information of ongoing colorectal cancer trials is frequently used and the majority of users is satisfied. The application provides easy to access and up-to-date information on ongoing clinical trials and is potentially useful to medical professionals in their busy daily   practices. These results warrant the development of an application including all registered clinical trials in which users can select their own trials of interest, including all diseases and specialties.